Crawl web

ToolsCrawl website... will download a website to local disc, search a website for useful info or both

It will crawl all webpages it can find below the URL you give it

If there are pages below this URL that may not be reachable from the top level you can add extra seed URLs

As well as the predefined patterns you can also search for arbitrary regular expressions

It's intentionally a dumb, impolite webcrawler, as it's designed for searching malign websites. It ignores the robots.txt file designed to tell webcrawlers where they're allowed to look. It ignores the META tags that do the same thing.

If the website includes self-referential cgi scripts designed to trap webcrawlers in an endless recursion they'll trap this one